System and method for evaluating job candidates

ABSTRACT

Human resource system for collecting and analyzing survey data from reference providers identified by a job candidate for use by an employer. The system sends an electronic communication to the reference providers to request completion of survey questions and electronically receives the survey data from the reference providers, preferably through webpages. An analysis module combines the received survey data from the reference providers and generates a confidential candidate report for an employer which excludes identification of any ratings or comments by any reference providers. The system also generates customized interview probe questions for use during job interviews and coaching tactics for use after the hiring, based on the weak areas that have been identified from the completed surveys in order to assist the hiring manager to bring the new hires up to speed quickly and effectively.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/275,633, filed on May 12, 2014, which is a continuation of U.S. Pat.No. 8,721,340, issued on May 13, 2014, which is a continuation of U.S.patent application Ser. No. 10/912,433, filed on Aug. 4, 2004, whichclaims priority to U.S. Provisional Patent Application No. 60/492,457,filed on Aug. 4, 2003, which are incorporated herein by reference.

FIELD OF THE INVENTION

This invention relates to human resource management system, and moreparticularly to a system for collecting and analyzing information fromreferences identified by job candidates.

BACKGROUND OF THE INVENTION

It has been estimated that errors in hiring cost companies more than 50billion dollars per year in lost revenue, decreased productivity,squandered training expenses, legal liabilities, high turnover and otherundesirable consequences. The result, in spite of the dramatic increasein the use of background checks and psychological profiles by manyorganizations, and the universality of selection interviews, is thatannual turnover in U.S. companies of over 5,000 employees continues tobe 25%. According to experts in the field, nearly 80% of turnover iscaused by poor selection decisions.

Therefore, background checks have become even more important than in thepast. One part of the background check, and more generally the hiringprocess, is the gathering of information from references, that is thoseindividuals identified by a job candidate as being knowledgeable aboutthe candidate's character and qualifications.

Unfortunately, traditional reference checking methods such as telephoneinterviews are very costly and time-consuming, require extensivetraining for interviewers and generally do not yield useful informationdue to lack of precision, lack of confidentiality and possible errors offiltering, amplification and interpretations by the intermediaries. Mostimportant, perhaps, is the fact that in today's litigious society,business and professional reference givers are usually unwilling toprovide more than basic information such as employment dates andpositions held. Such information has little value in helping companiesmake effective hiring decisions.

Another problem with the conventional reference checking is that it'sdone very late in the hiring process, which is typically done after thecandidate is hired. Ideally, it should be done during the screening andselection process. Moreover, the conventional background checkingprovides no guidance for the hiring manager to further explore areas ofweakness in the candidate during the hiring process.

Therefore, it would be desirable to provide a more effective andinexpensive system and method for collecting and evaluating informationprovided by reference providers for job candidates. It would be alsodesirable to provide such a system that is substantially automated andthat is used early in the hiring process.

SUMMARY OF THE DISCLOSURE

A system for collecting and analyzing survey data from referenceproviders identified by a job candidate for use by an employer isprovided. The system includes a candidate database that stores surveydata which are provided by the reference providers. A collection modulerunning in the system sends an electronic communication to the referenceproviders requesting them to complete the survey questions andelectronically receives the survey data. The electronic communicationpreferably contains a URL link that takes the reference provider to adynamically generated webpage through which the survey data are entered.

An analysis module running in the system combines the received surveydata from the reference providers and generates a candidate report. Inone aspect, the candidate report is a confidential report which excludesidentification of any ratings or comments by any reference providers. Inanother aspect, the system also generates customized interview probequestions for use during job interviews and coaching tactics for useafter the hiring, based on the weak areas that have been identified fromthe completed surveys in order to assist the hiring manager to bring thenew hires up to speed quickly and effectively.

The human resource system provides substantially automated collectionand analysis which is inexpensive and yet accurate and useful.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a job candidate evaluation system accordingto the present invention.

FIG. 2 is a simplified process flow of a candidate evaluation processaccording to the present invention.

FIG. 3 illustrates a detailed process flow of a purchase and set-up stepof FIG. 2.

FIG. 4 illustrates a detailed process flow of a candidate and referenceidentification step of FIG. 2.

FIG. 5 illustrates a detailed process flow of a collection step of FIG.2.

FIG. 6 illustrates a detailed process flow of an analysis step of FIG.2.

FIG. 7 illustrates a detailed process flow of a continuous update stepof FIG. 2.

FIGS. 8A and 8B illustrate a sample survey form.

FIG. 9 is a sample email sent to a job candidate.

FIG. 10 is a sample email sent to each reference identified by the jobcandidate.

FIGS. 11A-11D illustrate a sample candidate report for use by a hiringmanager.

FIGS. 12A-12B illustrate a sample set of interview questions for use bythe hiring manager in a subsequent interview with the job candidate.

FIGS. 13A-13C illustrate a sample set of coaching tactics for use by thehiring manager after the job candidate is hired.

FIG. 14 illustrates a group report which ranks multiple job candidates.

FIGS. 15A and 15B illustrate a sample vendor report that evaluatesvendors supplying goods and services to a company.

FIG. 16 illustrates a sample report that shows the correlation betweensurvey questions/competencies and performance of hired candidates.

DETAILED DESCRIPTION OF THE INVENTION

As an overview, the present invention uses a computer network such asthe Internet and the resources of the network including emails andwebpages to set up initial survey questions, send out emails toreferences identified by job candidates, collect confidentialcompetency-based survey information via webpages from the identifiedreferences, analyze the collected information and generate candidatereports for use by a hiring manager. The system has integrated thescreening and selection process with a competency based survey databasethat allows the comparative review of reference information against oneor more candidates, the company's own employees, the industry or othernormalized database by job type, organization or company competency.

Referring now to FIG. 1, a job candidate evaluation system 1 of thepresent invention involves a number of computers 10, 15 that areconnected to each other through a computer network such as the Internet.The computers 10, 15 of the system 1 cooperate with each other toprovide comprehensive collection and analysis of reference informationthat are made through the network 2. Computers 15 are similar to thecomputer 10, with the exception of some of the databases and softwaremodules.

As illustrated in FIG. 1, the computer 10 is connected to the Internet 2through, for example, an 110 interface 12, such as for a LAN, WAN, orfiber optic, wireless or cable link, which receives information from andsends information to other computers 15. The computer 10 is alsoconnected to a keyboard 14 for controlling the computer.

The computer 10 includes, for example, memory storage 16, processor(CPU) 18, program storage 20, and data storage 22, all commonlyconnected to each other through a bus 24. The program storage 20 stores,among others, software programs such as set-up module 26, collectionmodule 28, and analysis module 30 as will be explained in detail laterherein. The data storage 22 stores, among others, candidate database 34,benchmark database 36 and survey database 318, all preferably stored ina relational database that relates all of the databases stored in thedata storage. Any of the software program modules in the program storage20 and data from the data storage 22 are transferred to the memory 32 asneeded and is executed by the processor 18.

The computer 10 can be any computer such as a WINDOWS-based orUNIX-based personal computer, server, workstation or a mainframe, or acombination thereof. While the computer 10 is illustrated as a singlecomputer unit for purposes of clarity, persons of ordinary skill in theart will appreciate that the system may comprise a group of computerswhich can be scaled depending on the processing load and database sizeand which can be remotely located to provide localized non-stop service.

FIG. 2 illustrates a high level process flow of the evaluation processaccording to the present invention. In step 42, a client company sets upan order for a job candidate or multiple candidates, and prepares one ormore surveys for use in the hiring process as will be explained indetail with reference to FIG. 3. This step is executed by the set-upmodule 26. In the same step, the order for the job candidates is done bypurchasing a certain number of candidate reports and specifying thepurchase information. Purchase of one report provides unlimitedgeneration of reports for each job candidate until the time of hire orrejection.

In step 44, the hiring manager identifies a job candidate and receivesinformation about the references or reference providers that areidentified by the job candidate, which include an email address for eachreference as will be explained in more detail with reference to FIG. 4.A reference provider should be someone who has worked extensively withthe job candidate in the past which include customers, supervisors, andpeers. Step 44 is executed by the collection module 28.

In step 46, which is also executed by the collection module 28, thesystem 1 sends emails to all of the references that were identified bythe job candidate. The email requests each reference to fill out thesurvey prepared by the hiring manager. The survey information is thencollected from the identified references through web pages and stored inthe survey database 38. Step 46 is more fully explained with referenceto FIG. 5.

In step 48, executed by the analysis module 30, the system 1 analyzesthe collected information and generates a report that includes theoverall assessment of the candidate's competency in each of the severalcompetency areas and includes any comments supplied by the references.Competency is a well known concept that represents a particularcharacteristic of an individual or organization performing a task,function or project at a particular point in time that leads tosuccessful performance. The report can be a final report with allsurveys completed by the references, or it can be a real time interimreport with analysis of partially completed survey information which canbe requested by the hiring manager at any time. Based on the analysis,the system 1 also generates “interview probes” for those areas where thecandidate did not score as highly as others, a sample of which is shownin FIGS. 12A-12B. The probes guide an interviewer to obtain moreinformation about the candidate's level of accomplishments or experiencewith regard to specific lower scoring competencies. The report may alsoinclude coaching tactics to manage and develop the candidate assumingthe candidate is hired, a sample of which is shown in FIGS. 13A-13C.Like the interview probes, the coaching tactics are also based onanalysis of those areas where the candidate did not score as highly asothers. The coaching tactics are suggested “micro-behaviors” that thehiring manager can use to help the candidate to develop his strengths inlower-scoring competencies. Step 48 is more fully explained withreference to FIG. 6.

In step 50, the system 1 continuously monitors the job candidates afterthe hiring process. The system tracks the progress of the hiredcandidates and collects additional data such as performance levels ofthe hired candidates. The additional data for all the candidates arethen analyzed to generate additional reports containing the correlationbetween various competencies and high retention/performance. The reportsare preferably graphical in nature and graphically illustrate thecompetencies that are most closely correlated with the highretention/performance of the candidates. The reports can also becustomized by a user to specify whether the correlation is based onposition-specific, company-wide or industry-wide benchmarks as will beexplained in detail with reference to FIG. 7.

FIG. 3 illustrates a detailed process flow of the purchase and set-upstep 42 of FIG. 2.

In step 52, a client company uses an Internet enabled computer 15 toaccess web pages of the system 1 through the Internet 2. The web pagesare generated by a conventional database web page generating engine suchas PHP (Hypertext Preprocessor) in conjunction with a relationaldatabase program stored in the program storage 20 and the web engine isexecuted by the processor 18. The Internet enabled computer 15 isequipped with a web browser capable of handling forms.

In step 54, order information such as the client company's address andcontact information of various hiring managers working for the clientcompany are filled out in the web page form that was generated by thecomputer 10. In step 56, purchase information such as the number ofreports purchased and credit card data are also entered through the webpage. The data entered by the client company are stored in the datastorage 22. In step 58, the credit card information provided by theclient company is verified and in step 60 a client record is created ina client database in the storage 22 with the contact and purchaseinformation. The client record includes an allocation of reports tospecific hiring managers and the user id and password for each hiringmanager. In step 62, the computer 10 generates a confirmation messageconfirming the number of reports purchased and the set-up of the clientcompany in the system 1.

As can be appreciated by persons of ordinary skill in the art, the abovesteps 52-62 are optional and can be omitted by using a billingarrangement where the client company is billed on a periodic basis forthe candidate evaluation services that have been rendered.

FIG. 4 illustrates a detailed process flow of the candidate andreference identification step 44 of FIG. 2. When a hiring manager needsto fill a particular position, the manager accesses the computer 10through a web browser. In step 66, the hiring manager designates asurvey that is to be used for that position. The manager can choose froma set of pre-designed or pre-selected surveys stored in the data storage22, design his own by selecting survey questions from an existing set ofquestions stored in the data storage, create his own set of questions,or modify an existing survey. The questions are stored in a master tablein data storage 22. Each survey also has a corresponding record in thedatabase, which points to the questions in the master table that areincluded for that survey. Each survey question relates to a specificjob-related and validated competency, selected from a bank ofcompetencies that have been derived, tested and validated fromexperience and research.

FIGS. 8A-8B illustrate a sample survey that is used for a managementposition. The survey of FIG. 8A contains 16 questions that relate tovarious competencies that are known to be important for a managementposition. For example, the first three questions relate to a competencyknown as “Managing the Business”. Each of the 16 questions requires thereference to select a value of “1” through “7” by clicking on anappropriate radio button. The survey also contains two comment boxes asshown in FIG. 8B. It includes one for describing the candidate'sstrengths and one for describing the candidate's weaknesses.

Once a particular survey is selected or created, the hiring managerenters the job candidates' information in step 68 through the computer15 and sets the required minimum number of references that must beprovided by each job candidate. The entered information is stored by thecomputer 10 in the candidate database 34 of the data storage 22. In step68, the computer 10 also generates a unique 16 character alphanumericidentifier for that job candidate which is also stored in the candidatedatabase 34. The alphanumeric identifiers are used for security purposessince they ensure that only known and authorized job candidates canenter or access the information in the system 1.

In step 70, the computer dynamically generates a web page asking whetherthe reference information will be provided by the job candidate. If thehiring manager answers yes, the computer 10 in step 72 generates andsend an email message to the job candidate with a URL link to adynamically generated web page and requesting the job candidate to clickon the link to provide information on the references he chooses. Asample email to a job candidate is shown in FIG. 9.

In step 74, the job candidate receives the email and accesses the webpage generated by the computer 10 by clicking on the link provided inthe email. In response, the computer transmits through the Internet 2 asample of the survey questions for display on the candidate's computer15 along with a dynamically generated web page form to provideinformation on the references. The sample survey questions assist thecandidate in determining which individuals would be appropriatereferences. In step 76, the job candidate enters via the web page formthe names, email addresses and relationship of the references. Therelationship field only allows “Business” or “Professional” as“Personal” references tend to give scores that are severely skewedtowards the positive, and may not have specific knowledge about thejob-related competencies of the candidate. The candidate also indicatesthe dates and location of the relevant employment. The candidate thensubmits the form to the computer 10.

The collection module 28 of the system 10 then verifies that each emailaddress is in a valid format and that there are no duplications. As partof the validity check, the collection module 28 checks the domainportion of each email address against the registered location usingindustry standard databases (WHOIS) to provide the hiring manageradditional information if needed. Once the candidate is determined tohave submitted a valid list, the computer 10 stores the data onreferences in the candidate database 34.

If the answer to step 70 is no, however, then the hiring manager alreadyhas the information of references. That information is entered by thehiring manager in step 78. The same type of data checking that areperformed in step 76 is also performed in step 78 to ensure that noerrors are made.

FIG. 5 illustrates a detailed process flow of the collection step ofFIG. 2. In step 80, the computer 10 generates a unique identifier foreach reference and send an email message to each reference explainingthe purpose of the email and directing the reference to click on a URLlink to a dynamically generated web page. The unique identifier isgenerally used internally to uniquely identify the reference within thesystem 1. A sample email to each reference is shown in FIG. 10. Thesample email contains a statement that the operator of the system 1 willmaintain strict confidentiality of responses provided by the referencesand that their responses will be aggregated and analyzed so that all ofthe information generated in a report for the hiring manager will beconfidential. This statement is important because it encourages thereferences to provide more honest responses.

In step 82, the email is received by the computer 15 of the reference.In step 84, the reference accesses the web page generated by thecomputer 10 by clicking on the link provided in the email. In response,the computer transmits through the Internet 2 a dynamically generatedweb page form for display on the reference's computer 15 along withinstructions on how to properly complete the form, a sample of which isshown in FIGS. 8A-8B.

In step 86, the reference enters via the web page form answers to thequestions in the survey. For each question, the reference indicates thelevel of competency possessed by the job candidate using a seven-pointscale. The reference is also shown the employment information submittedby the job candidate and is asked to confirm whether the information isaccurate. The reference then submits the completed form to the computer10 in step 88.

In step 90, the collection module 28 of the system 10 stores the surveydata in the survey database 38. In step 92, the collection module 28determines whether there is a sufficient number of completed surveys toprovide a meaningful report to the hiring manager. For example, in onecase that requires seven references, four references might be consideredto provide a meaningful report. If no, then the collection module 28waits for additional surveys to be completed. If yes, however, step 94is executed.

Alternatively, the hiring manager has three additional options at thisstage. The first option is to override the minimum number of completedsurveys and to request an interim candidate report reports regardless. Asecond option is to set a predetermined time period from the jobcandidate identifies the references and checking to see whether thepredetermined time period has passed. If it has, then step 94 isexecuted. The third option is to simply allow the hiring manger to closethe job candidate's record. That option may be convenient in situationssuch as the job candidate voluntarily withdrawing from the job opening.

In step 94, an email to the hiring manager is generated to let him knowthat at least an interim report is available, which email is received byhim in step 98. In step 96, reminder emails are sent to those referencesthat have not provided the survey data within a predetermined timeperiod.

FIG. 6 illustrates a detailed process flow of the analysis step of FIG.2. In step 100, the hiring manager accesses the web page generated bythe computer 10 by either clicking on the link provided in the email orlogging on to the website of the computer 10 independent of the email.At this stage, the analysis module 30 displays a “dashboard” for thecurrent status of the job opening. The status includes the state of eachcandidate's progress for the completion of the reference information. Instep 102, a request for report on a particular candidate is made.

In response, the analysis module 30 in step 104 analyzes the receivedsurvey data stored in the survey database 38 and generates a candidatereport. A sample report is shown in FIGS. 11A-11D.

As can be seen in FIG. 11A, the report includes a list of references,email addresses, identification of company and relation to the jobcandidate, dates worked by the candidate and whether the survey wascompleted. The report also notes any changes or discrepancies betweenthe information provided by the candidate and the reference. Forexample, reference named “Roger Brown” reported that the dates worked bythe candidate of “Feb. 2000 To Feb. 2003” is different from “Jan. 2000To Feb. 2003”. FIG. 11B explains the score for each competency in FIG.11C which is based on benchmark scores that are stored in the benchmarkdatabase 36. The benchmark scores represent competencies that are storedon an industry-wide basis, company-wide basis or company-specific jobposition type basis.

As seen in FIG. 11C, scores for each question are averaged and convertedto “very low” to a “very high” score. The scores to questions that arerelated to the same competency are averaged into the same “very low” toa “very high” score. For example, the average scores for questions 1-3have been converted to a “Medium”, “High” and “Medium”, respectively.Also, the three questions are grouped into the competency of “Managingthe business” and the average scores for each of questions 1-3 areaverages and converted to the score of “Medium”.

One third of the questions where the candidate has received the lowestraw numerical averages are weak areas the analysis module has identifiedand are indicated using asterisks which are used as the basis forgenerating interview probe questions and coaching tactics as discussedbelow in step 106. Alternatively, the weak areas are identified bycomparing an averaged score for each question against a benchmark scorefrom the benchmark database 36 and those scores that fall below thebenchmark score by a predetermined amount are identified as the weakareas and are indicated as such using asterisks.

To further make the report useful, it may contain three additionalcolumns: Company Comparison, Industry Comparison and Overall Rating. TheCompany Comparison and Industry Comparison correspond to the benchmarkscores on company-wide job type basis, and industry-wide job type basis.The Overall Rating is derived from taking the average of the raw scoresfrom each normalized database for that job type for the company,industry or other organization and the average of responses from thecandidate's reference providers. This score is converted to a verbaldescriptor from very-low to very-high based on lookup table for thatvalue. For example a 4.2-5.9 inclusive, could return a “High”, eachdescriptor range can be set based on selection standards.

As shown in FIG. 11C, questions 3, 8, 11 and 16 are highlighted using arectangular box. The highlighted questions mean that those questions aremost closely associated with high performance and retention of jobcandidates that have been hired which are based on a statisticalanalysis of performance data of the hired candidates as will beexplained in detail with reference to FIG. 7.

The report also contains an overall average score (“High” in FIG. 11C)which is an average of scores from all the references for all of thequestions. Alternatively, the analysis module 30 can use the correlationdata from the continuous update step 50 of FIG. 2 and generate anoverall score on a weighted scale in which the weight used for eachquestion or competency is based on the correlation to the performancedata with higher weight being used for higher correlation.

FIG. 11D contains the “Strengths” and “Could Improve” comments providedby the references. It is important to note that the candidate reportmaintains strict confidentiality of responses provided by thereferences. In other words, the candidate report decouples the referenceindividuals from the responses the reference individuals provide so asto provide anonymity of the reference individuals from the hiringcompany/hiring manager. This is important as it encourages thereferences to provide more candid responses.

FIG. 14 illustrates a group report which ranks multiple candidates. Foreach candidate, the group report includes averaged scores for eachquestion, averaged score for each competency, and an overall summaryscore. The group report also includes questions 3, 8 and 16 which arehighlighted using a rectangular box. The highlighted questions mean thatthose questions are most closely associated with high performance andretention of job candidates that have been hired which are based on astatistical analysis of performance data of the hired candidates.

In step 106, the hiring manager, after reviewing the report, determineswhether to continue with the hiring process for the candidate. If theanswer is no, then the analysis step 48 ends at step 111. If the answeris yes, the analysis step 48 continues with step 108. In step 108, theanalysis module generates interview questions (see FIGS. 12A-12B) basedon the identified weak areas in the report. Specifically, the interviewquestions are associated with the questions in the survey and are storedin the data storage 22. For each weak area, the analysis moduleretrieves those interview questions that are associated with thequestions that correspond to the weak areas.

For example, the report in FIG. 11C has identified questions 1, 2, 7, 9,12 and 13 as the weak areas. For those questions, the analysis module 30retrieves associated interview questions from the data storage 22 asshown in FIGS. 12A-12B.

In step 110, the hiring manager, after having interviewed the candidate,determines whether to hire the candidate. If the decision is no, thenthe analysis step 48 ends. If the decision is a yes, then the analysismodule in step 112 generates coaching suggestions that allow theemployer to improve the identified weak areas after the candidate ishired. Like the interview questions, the coaching suggestions areassociated with the questions in the survey and are stored in the datastorage 22. For each weak area, the analysis module 30 retrieves thosecoaching suggestions that are associated with the questions thatcorrespond to the weak areas as shown in FIGS. 13A-13C.

It is important to note that the steps of survey collection 46 andanalysis 48 can be used in an iterative process to screen out jobcandidates. For example, a hiring manager might use a simple surveycontaining 4 questions against 100 job candidates to narrow the listdown to 10 candidates, then design a new survey containing 16 questionsto narrow the 10 candidates down to 3 candidates, and then designanother survey containing 20 questions to select one candidate for hire.Accordingly, the present invention can be used to both as a screeningtool and a selection tool.

FIG. 7 illustrates a detailed process flow of the continuous update stepof FIG. 2 which is also part of the analysis module 30. In step 120, thecomputer 10 waits for a predetermined time period after the finalcandidate report was generated. In the embodiment shown, thepredetermined time period is one month. In step 122, the computer 10prepares and sends an email to the hiring manager with a URL link to adynamically generated web page. The web page asks whether a particularjob candidate is hired. Alternatively, the email may include two linksasking the hiring manager to click on one link if the candidate washired and to click on the other link is the candidate was not hired.

In step 124, the computer 10 receives the response of the hiring managerand determines whether the job candidate was hired. If no, then thatfact is noted and stored in the candidate database 34 in step 126 forlater analysis and reporting. If the candidate was hired, then controlpasses to step 128. At step 128, the computer 10 waits for apredetermined time period after the final candidate report wasgenerated. In the embodiment shown, the predetermined time period is oneyear from the final report. In step 130, the computer 10 prepares andsends an email to the hiring manager with a URL link to a dynamicallygenerated survey web page. The survey web page asks retention andperformance information. In the embodiment shown, the survey web pageasks two questions: (1) is the candidate still employed; and (2) howwell the candidate has performed based on a survey containing multiplequestions or based on a single question on a predefined scale, e.g.,scale of 1-10. In step 126, the response to the two questions from thehiring manager are stored in the candidate database 34 for lateranalysis and reporting.

In step 132, the analysis module 30 analyzes the hiring data stored inthe candidate database 34. Specifically, the retention and performancedata are statistically correlated with the various scores received bythat candidate in the surveys to identify the questions where highratings are most closely associated with high performance and retention.The correlation can be calculated on an industry-wide position typebasis, on a company-wide basis without regard to the position type or ona company-wide position type basis. Thus, when the candidate reportssuch as shown in FIG. 11C are generated, the questions where highratings are most closely associated with high performance and retentionare graphically indicated based on the latest data accumulated up tothat point.

FIG. 16 illustrates a sample report that shows the correlation betweensurvey questions/competencies and performance of hired candidates. Asshown, the report includes raw correlation scores and correspondingratings. For example, question 3 has a raw score of 4.7 and a “VeryHigh” rating.

These reports can be used by the hiring manager to continuously improvethe survey. For example, the hiring manager can choose to delete the twoquestions that have the lowest correlation to the performance data andadd two new questions from the competency area that has the highestcorrelation to the performance data. In the example shown, the twolowest scoring questions are questions 1 and 2, and the competency areahaving the highest score of 4.66 is “Teamwork”.

Once step 132 is executed, control passes to step 128 where the computerwaits a programmable amount of time usually 6 to 15 months to repeat thesteps 130 to 132 to continuously monitor the performance of the jobcandidates that have been hired in order to continuously improve thesurvey questions and competency categories.

Application of the principles of the present invention are many. Forexample, principles of the survey selection, collection of responses andanalysis of the responses can be used to evaluate a large number ofvendors who supply products and services to a company through a group ofbuyers working for the company. The buyers for the company are acting as“reference providers”. A sample vendor report as shown in FIGS. 15A-15Bcan be used to better manage the large number of vendors.

The foregoing specific embodiments represent just some of the ways ofpracticing the present invention. Many other embodiments are possiblewithin the spirit of the invention. Accordingly, the scope of theinvention is not limited to the foregoing specification, but instead isgiven by the appended claims along with their full range of equivalents.

What is claimed is:
 1. A computer-implemented method being implementedby an improved computer system programmed for reference checking inaccordance with the computer-implemented method, comprising: storing, bythe improved computer system programmed for reference checking, in atleast one database, a plurality of position specific survey questionsspecific to competency skills related to performance of a plurality ofpositions; wherein each response received, by the improved computersystem programmed for reference checking, to each position specificsurvey question corresponds to a number on a numerical extent scale;receiving, by the improved computer system programmed for referencechecking, through at least one first computer programmed graphical userinterface, from at least one first employer, at least the followinginformation: i) job information, wherein the job information is relatedto at least one first position, and ii) an identity of each jobcandidate applying to be interviewed; wherein the at least one firstemployer requires references from a plurality of reference providers tobe received for each job candidate before the at least one firstemployer decides whether or not to conduct a job interview; wherein theimproved computer system programmed for reference checking isindependent from each job candidate and the at least one first employer;based on the job information received from the at least one firstemployer, selecting, by the improved computer system programmed forreference checking, from the at least one database, a subset of positionspecific survey questions being specific to competency skills related toperformance of the at least one first position; based on the selectingthe subset of position specific survey questions being specific tocompetency skills related to performance of the at least one firstposition, generating, by the improved computer system programmed forreference checking, at least one first position specific survey relatedto the at least one first position; wherein the at least one firstposition specific survey is configured for calculating an statisticalscore in each competency skills group of the competency skills based onstatistical analysis of numbers on the numerical extent scale acrossresponses received from the plurality of reference providers to thesubset of position specific survey questions so that the responses ofthe plurality of reference providers are confidential from employers andjob candidates; receiving, by the improved computer system programmedfor reference checking, through at least one second computer programmedgraphical user interface, from a first job candidate from a plurality ofjob candidates applying to be interviewed, first contact informationidentifying the plurality of reference providers, automaticallytransmitting, by the improved computer system programmed for referencechecking, at least one first personalized request to complete the atleast one first position specific survey to the first referenceprovider, wherein the at least one first personalized request comprises:i) a first URL link to access the at least one first position specificsurvey, and ii) information informing that responses obtained inresponse to the at least one first position specific survey are keptconfidential from the at least one first employer and the first jobcandidate; automatically transmitting, by the improved computer systemprogrammed for reference checking, at least one second personalizedrequest to complete the at least one first position specific survey tothe second reference provider, wherein the at least one secondpersonalized request comprises: i) a second URL link to access the atleast one first position specific survey, and ii) the informationinforming that the responses obtained in response to the at least onefirst position specific survey are kept confidential from the at leastone first employer and the first job candidate; automaticallytransmitting, by the improved computer system programmed for referencechecking, at least one third personalized request to complete the atleast one first position specific survey to the third referenceprovider, wherein the at least one third personalized request comprises:i) a third URL link to access the at least one first position specificsurvey, and ii) the information informing that the responses obtained inresponse to the at least one first position specific survey are keptconfidential from the at least one first employer and the first jobcandidate; causing to display, by the improved computer systemprogrammed for reference checking, through at least one third computergraphical user interface, the at least one first position specificsurvey to the first, the second and the third reference providersrespectively in response to the first, the second and the third URLlinks being respectively activated; respectively receiving, by theimproved computer system programmed for reference checking, from thefirst, the second and the third reference providers, the responses tothe at least one first position specific survey; calculating, by theimproved computer system programmed for reference checking, eachstatistical score of the first job candidate in each competency skillsgroup of the competency skills based on the responses, beingrespectively received from the first, the second and the third referenceproviders; generating, by the improved computer system programmed forreference checking, for the at least one first employer, at least onereference report related to the first job candidate, wherein the atleast one reference report comprises the statistical scores of the firstjob candidate in the competency skills groups of the at least one firstposition specific survey, wherein the statistical scores of the firstjob candidate being calculated based on the responses of the first, thesecond and the third reference providers, so as to maintain theconfidentiality of the responses from the at least one first employerand the first job candidate; wherein the statistical scores of the firstjob candidate are benchmarked against statistical scores of other jobcandidates who applied to a specific position that is at least similarto the at least one first position; and wherein the at least onereference report is configured to allow the at least one first employerto decide whether or not to conduct the job interview with the first jobcandidate who has applied to be interviewed.
 2. The computer-implementedmethod of claim 1, wherein the statistical scores of the first jobcandidate are benchmarked on a company-wide basis.
 3. Thecomputer-implemented method of claim 1, wherein the statistical scoresof the first job candidate are benchmarked on an industry-wide basis. 4.The computer-implemented method of claim 1, wherein the first contactinformation identifies a first email address, a second email address,and a third email address for the first, the second and the thirdreference providers, respectively; wherein the receiving the firstcontact information identifying the at least three reference providersfurther comprises: validating domain portions of the first, the second,and the third email addresses against registered Internet locations,respectively, based at least in part on at least one industry standarddatabase of the Internet locations; wherein the automaticallytransmitting the at least one first personalized request to complete theat least one first position survey to the first reference providercomprises: sending the at least one first personalized request to thevalidated first email address; wherein the automatically transmittingthe at least one second personalized request to complete the at leastone first position survey to the second reference provider comprises:sending the at least one second personalized request to the validatedsecond email address; and wherein the automatically transmitting the atleast one third personalized request to complete the at least one firstposition survey to the third reference provider comprises: sending theat least one third personalized request to the validated second emailaddress.
 5. The computer-implemented method of claim 1, wherein eachstatistical score of the first job candidate is an average score;wherein the statistical scores of the first job candidate are averagescores; and wherein the statistical analysis is based on averagingnumbers of the numerical extent scale across responses received fromreference providers to the plurality of survey questions.
 6. Thecomputer-implemented method of claim 2, wherein the statistical scoresof the first job candidate are average scores.
 7. Thecomputer-implemented method of claim 3, wherein the statistical scoresof the first job candidate are average scores.
 8. Thecomputer-implemented method of claim 1, wherein the job information atleast identifies whether the at least one first position involvesmanaging others.
 9. The computer-implemented method of claim 1, whereinthe computer-implemented method further comprises: automaticallyassigning, by the improved computer system programmed for referencechecking, a first unique identifier to a first reference provider of theplurality of reference providers; wherein the at least one firstpersonalized request is associated with the first unique identifier;automatically assigning, by the improved computer system programmed forreference checking, a second unique identifier to a second referenceprovider of the plurality of reference providers; wherein the at leastone second personalized request is associated with the second uniqueidentifier; automatically assigning, by the improved computer systemprogrammed for reference checking, a third unique identifier to a thirdreference provider of the plurality of reference providers; and whereinthe at least one third personalized request is associated with the thirdunique identifier.
 10. The computer-implemented method of claim 9,wherein the first URL link of the at least one first personalizedrequest is encoded with the first unique identifier; wherein the secondURL link of the at least one first personalized request is encoded withthe second unique identifier; and wherein the third URL link of the atleast one first personalized request is encoded with the third uniqueidentifier.
 11. An improved computer system programmed for referencechecking, comprising: at least one specialize computer machine,comprising: a non-transient memory having at least one region forstoring particular computer executable program code; and at least oneprocessor for executing the particular program code stored in thememory, wherein the particular program code is configured to at leastperform the following operations upon the execution: storing in at leastone database, a plurality of position specific survey questions specificto competency skills related to performance of a plurality of positions;wherein each response received, by the improved computer systemprogrammed for reference checking, to each position specific surveyquestion corresponds to a number on a numerical extent scale; receiving,through at least one first computer programmed graphical user interface,from at least one first employer, at least the following information: i)job information, wherein the job information is related to at least onefirst position, and ii) an identity of each job candidate applying to beinterviewed; wherein the at least one first employer requires referencesfrom a plurality of reference providers to be received for each jobcandidate before the at least one first employer decides whether or notto conduct a job interview; wherein the improved computer systemprogrammed for reference checking is independent from each job candidateand the at least one first employer; based on the job informationreceived from the at least one first employer, selecting, from the atleast one database, a subset of position specific survey questions beingspecific to competency skills related to performance of the at least onefirst position; based on the selecting the subset of position specificsurvey questions being specific to competency skills related toperformance of the at least one first position, generating at least onefirst position specific survey related to the at least one firstposition; wherein the at least one first position specific survey isconfigured for calculating an statistical score in each competencyskills group of the competency skills based on statistical analysis ofnumbers on the numerical extent scale across responses received from theplurality of reference providers to the subset of position specificsurvey questions so that the responses of the plurality of referenceproviders are confidential from employers and job candidates; receiving,through at least one second computer programmed graphical userinterface, from a first job candidate from a plurality of job candidatesapplying to be interviewed, first contact information identifying theplurality of reference providers, automatically transmitting at leastone first personalized request to complete the at least one firstposition specific survey to the first reference provider, wherein the atleast one first personalized request comprises: i) a first URL link toaccess the at least one first position specific survey, and ii)information informing that responses obtained in response to the atleast one first position specific survey are kept confidential from theat least one first employer and the first job candidate; automaticallytransmitting at least one second personalized request to complete the atleast one first position specific survey to the second referenceprovider, wherein the at least one second personalized requestcomprises: i) a second URL link to access the at least one firstposition specific survey, and ii) the information informing that theresponses obtained in response to the at least one first positionspecific survey are kept confidential from the at least one firstemployer and the first job candidate; automatically transmitting atleast one third personalized request to complete the at least one firstposition specific survey to the third reference provider, wherein the atleast one third personalized request comprises: i) a third URL link toaccess the at least one first position specific survey, and ii) theinformation informing that the responses obtained in response to the atleast one first position specific survey are kept confidential from theat least one first employer and the first job candidate; causing todisplay through at least one third computer graphical user interface,the at least one first position specific survey to the first, the secondand the third reference providers respectively in response to the first,the second and the third URL links being respectively; respectivelyreceiving, from the first, the second and the third reference providers,the responses to the at least one first position specific survey;calculating each statistical score of the first job candidate in eachcompetency skills group of the competency skills based on the responses,being respectively received from the first, the second and the thirdreference providers; generating, for the at least one first employer, atleast one reference report related to the first job candidate, whereinthe at least one reference report comprises the statistical scores ofthe first job candidate in the competency skills groups of the at leastone first position specific survey, wherein the statistical scores ofthe first job candidate being calculated based on the responses of thefirst, the second and the third reference providers, so as to maintainthe confidentiality of the responses from the at least one firstemployer and the first job candidate; wherein the statistical scores ofthe first job candidate are benchmarked against statistical scores ofother job candidates who applied to a specific position that is at leastsimilar to the at least one first position; and wherein the at least onereference report is configured to allow the at least one first employerto decide whether or not to conduct the job interview with the first jobcandidate who has applied to be interviewed.
 12. The improved computersystem programmed for reference checking of claim 11, wherein thestatistical scores of the first job candidate are benchmarked on acompany-wide basis.
 13. The improved computer system programmed forreference checking of claim 11, wherein the statistical scores of thefirst job candidate are benchmarked on an industry-wide basis.
 14. Theimproved computer system programmed for reference checking of claim 11,wherein the first contact information identifies a first email address,a second email address, and a third email address for the first, thesecond and the third reference providers, respectively; wherein thereceiving the first contact information identifying the at least threereference providers further comprises: validating domain portions of thefirst, the second, and the third email addresses against registeredInternet locations, respectively, based at least in part on at least oneindustry standard database of the Internet locations; wherein theautomatically transmitting the at least one first personalized requestto complete the at least one first position survey to the firstreference provider comprises: sending the at least one firstpersonalized request to the validated first email address; wherein theautomatically transmitting the at least one second personalized requestto complete the at least one first position survey to the secondreference provider comprises: sending the at least one secondpersonalized request to the validated second email address; and whereinthe automatically transmitting the at least one third personalizedrequest to complete the at least one first position survey to the thirdreference provider comprises: sending the at least one thirdpersonalized request to the validated second email address.
 15. Theimproved computer system programmed for reference checking of claim 11,wherein each statistical score of the first job candidate is an averagescore; wherein the statistical scores of the first job candidate areaverage scores; and wherein the statistical analysis is based onaveraging numbers of the numerical extent scale across responsesreceived from reference providers to the plurality of survey questions.16. The improved computer system programmed for reference checking ofclaim 12, wherein the statistical scores of the first job candidate areaverage scores.
 17. The improved computer system programmed forreference checking of claim 13, wherein the statistical scores of thefirst job candidate are average scores.
 18. The improved computer systemprogrammed for reference checking of claim 11, wherein the jobinformation at least identifies whether the at least one first positioninvolves managing others.
 19. The improved computer system programmedfor reference checking of claim 11, wherein the particular program codeis further configured to at least perform the following operations uponthe execution: automatically assigning a first unique identifier to afirst reference provider of the plurality of reference providers;wherein the at least one first personalized request is associated withthe first unique identifier; automatically assigning a second uniqueidentifier to a second reference provider of the plurality of referenceproviders; wherein the at least one second personalized request isassociated with the second unique identifier; automatically assigning athird unique identifier to a third reference provider of the pluralityof reference providers; and wherein the at least one third personalizedrequest is associated with the third unique identifier.
 20. The improvedcomputer system programmed for reference checking of claim 19, whereinthe first URL link of the at least one first personalized request isencoded with the first unique identifier; wherein the second URL link ofthe at least one first personalized request is encoded with the secondunique identifier; and wherein the third URL link of the at least onefirst personalized request is encoded with the third unique identifier.